Can we insist students reach proficiency on homework? Yes!

This study involved a comparison between the conditions of Mastery Mode against NonMastery. In the Mastery Mode, selected problems were mastery problems. A student, who got an incorrect answer, went into the Mastery Mode in which he/she had to get three consecutive problems correct testing the same skill. Although this process took long, it forced the student to master the subject matter. In the Non-Mastery Mode, the students were given two chances for each problem while no tutoring was provided. It was observed through the results that there was a significant difference between the Mastery Mode and Non-Mastery with the p-value of 0.003. The effect of the study was towards the mastery mode and students learned significantly more in this condition with an effect size of 0.52. The implication of this experiment is that mastery learning is an efficient technique which can be incorporated in homework to make students put effort in learning the content at different time intervals, thus, increasing the overall learning gain. Introduction America is constantly searching for innovations that will drastically increase learning in students. The idea of mastery learning is at least 70 years old (Davis & Sorrell, 1995). Bloom studied and found positive results on the effectiveness of mastery learning. In Blooms’ studies, students kept working on a topic until they reached some threshold. The results of the experiment were positive with mastery learning showing significant effect in the learning of students. Many computer systems are already available which provide some type of mastery learning (Corbett and Anderson, 1995, Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A.) The state of Maine runs a laptop program which leases laptop to all the middle school students in the state. The program started in 2001 and has been running ever since. Silvernail & Buffington ran a study to see the impact of laptop use in the NAEP scores of the students between 2002 and 2009. While NAEP scores have risen 10 points from 2000-2009 across the nation, NAEP scores in Maine have only gone up by 5 points. The report states that only half of the mathematics teachers self-reported that they were using their laptops for instruction when Silvernail and Buffington began their study (their study was one in which teachers were given 210 hours of professional development in using the laptops to help math classes). It should be pointed out that the state bought the hardware but did not buy any math educational software. Suppose students across Maine got computer supported homework, would it be good to add a mastery component or is such drill and kill work not likely to lead to more learning? That is an issue we are trying to investigate in this study. Currently there is a good deal of literature that is skeptical of the value of education based on computers. Congress mandated a large study conducted by RAND (Devin, 2004) to see if a handful of math and reading computer products produced real learning results as measured by state tests scores. It was a high quality evaluation that cost millions of dollars. It had random assignment of interventions to schools, and was well powered enough to detect small effects but did not find any conclusive result. In another study RAND (McArthur & Stasz, 1990) analyzed the impact of another computer tutor and found a reliable negative effect for the condition with the cognitive tutor. Are these pieces of software really all that useful? It could be said that in both studies they reported that the teachers did not use them as much as the experiment expected them to use, so maybe that explains the results. So what should the governor of Maine do? Stop the expensive laptop program? Or invest in good technology that the math teachers can use to supplement their teaching? We are not sure, but we wanted to ask a more limited question. Assuming a teacher assigned homework on the computer, would students learn more if they were forced to master the topics? There is a good deal of skepticism about homework. In a TIME Magazine article, “The Myth about Homework”, Claudia Wallis concludes that “more homework brings diminishing returns” and that it does not improve academic achievement. Alfie Kohn, a widely known writer for human behavior and parenting, shares a similar viewpoint in his book The Homework Myth in which he questions the benefits of overloading the students with homework. In “The End of Homework: How Homework Disrupts families, Overburdens Children, and Limits Learning” Kralovec and Buell (2000) argue that homework has very limited benefits and are skeptical about overburdening students, who already have a lot of work, with repetitive busy work. In “The Case against Homework: How Homework Is Hurting Our Children and What We Can Do About It” (Bennett & Kalish, 2007) the authors argue that there is no evidence that home work helps elementary school students achieve academic success and little evidence that it helps older students. A meta-analysis by Cooper, Robinson & Patall (2006) found a mild effect of homework, but report that there can be diminishing positive effect. Could Mastery Learning on computers be just the sort of reparative activity that might show such effects? In this experiment we try to investigate if such repetitive learning method will have positive effect in learning or will it result in negative learning. The ASSISTment System The ASSISTment System is a web-based tutoring system, capable of offering instructions to students while providing detailed evaluations of their performance to teachers. The system integrates assistance and assessment methods to efficiently tutor students in mathematics and is being used by several middle school and high school teachers throughout Massachusetts. Teachers may use this system as part of their coursework to assist students in learning while also obtaining detailed reports on class performance or on individual students. Teachers may then identify difficulties students may be facing to tailor their instructions to be more effective. The system is free to use and is supported from grants from the U.S. Department of Education and the National Science Foundation. One of the primary design goals of the system is to efficiently tutor students using the process of formative assessment. Influenced by earlier systems such as Cognitive Tutors (Anderson et al. 1995), the www.ASSISTment.org System tutors students as they are progressing through an assignment on problems they get wrong. This process of providing feedback and changing tutoring methods without the teacher having to intervene aims towards improving the effectiveness of student learning time. The tool is available online and schools can access it for free. A study conducted in 2009 by Razzaq et al showed that students were reliably learning through the system. A second study (Koedigner, McLaughin, and Heffernan, in press) showed in a quasi-experimental design that students assigned to the ASSISTments condition had higher gains on their state tests, but the sample had only 4 schools in total. Various tutoring methods are available within the system for assisting students learning. One of these methods is to give hints to students which may provide general ideas of how to approach the problem or may provide a key step needed for the problem. Another method called tutored problem solving breaks a harder problem into simpler ones, each of which can be viewed as a step needed to solve the original problem. Usually, a combination of both these methods is used. Usually, when a student answers the initial problem incorrectly, the system breaks the problem down into individual steps, and asks the student to answer each step. Hints are provided to assure the student can eventually answer these steps and that he\she does not become stuck. Figure 2 demonstrates this and shows the use of tutoring that involves basic algebra and the properties of supplementary angles. Another form of tutoring used for the purpose of this experiment is the Mastery Learning method of instruction. In this method, a student must show mastery on questions of a given skill before proceeding. To do this, they must correctly answer a given number of consecutive questions correctly on their first attempt without asking for help. We refer to this specific number of questions as the mastery limit and remark that it is set by the teacher on each assignment. The problems presented to the student are designed to be one on skill, similar to each other and are referred to as morphs of one another. For each of these problems within the Mastery Learning method, different forms of tutoring such as hints or tutored problem solving can be used to tutor the student on the problem. Figure 1 shows an example of a problem in a Mastery Learning assignment. If a student is not able to answer the problem shown in Figure 1 without the help of any hints, another problem is given that focuses on the same skill. Figure 3 shows an example of a morph of the problem shown in Figure 1. A student will keep receiving morphs of the problem until the student can demonstrate competency in the skill being tested on by answering the Mastery Limit number of problems correctly. Figure 1: A problem with five hints focusing on growth factors in exponential equations. A student does not receive a hint until they ask for one. Figure 2: Tutoring mode showing tutored problem solving with hints on each sub-problem. Figure 3: A morph of the problem shown in Figure 2. Experiment This experiment focuses on analyzing the effectiveness of homework assigned to students following the guidelines of the concept of Mastery Learning over Regular homework. The control group consisted of students participating in Non-Mastery Mode condition, which represents Regular homework. The students in Non-Mastery Mode condition also did their homework online in the ASSISTments system, but did not receive any form of feedback. Setting and Participants The experiment was conducted on students in Eighth grade classes. The entire study was conducted using Web-based assignments through the ASSISTments system. Students were given homework assignments and later a posttest to measure their learning from the homework problems. The web-based homework assignments were completed by students at home using their home computers, while the students used computers present in classrooms for their posttest. The study comprised three classes which had a total number of 128 students. Each class contained a mix of below-average, average and above average students. Figure 4: A problem that references the CMP-Growing Growing Growing book. Content The content consisted of questions that tested the understanding of linear and exponential growth rates. The content for the primary problems used for this experiment was extracted from the textbook of the Connected Mathematics Project (CMP) curriculum-CMP Growing Growing Growing. The remaining problems consisted of morphs of problems from the textbook. Examples of original problem and morphs of the problems can be found in the Appendix. All the problems used in this experiment that are assigned as a part of the regular homework are problems recommended by the CMP curriculum. According to Cain (2002), CMP students performed significantly better in state level standardized tests than non-CMP students. Experimental Design To test the effectiveness of Mastery Learning homework the students in the three classes included in our study were each divided into two groups. The division was randomized based on the last name of students. Two types of homework assignments were designed for the treatment and control group of the study. The control group was assigned questions that directly referenced problems in the homework section of the CMP-GGG book. These problems did not contain any form of tutoring, i.e. no hints were provided. The student was simply given a second chance to answer the question before revealing the right answer. The student is considered to have incorrectly answered the question if he/she is not able to provide the right answer in the first attempt. The homework assignment of this condition was designed to simulate regular book homework given to students. One of the problems from the book used in this study is shown in Figure 5. The problem in Figure 4 refers to the problem in Figure 5. The students in the treatment group were also given questions that referenced problems in the CMP-GGG book. However, a student is not allowed to progress through an assignment by answering questions incorrectly. Instead of presenting the next problem in the sequence, the system informs the student they must demonstrate mastery on the topic before moving on to complete their homework. The system then presents morphs of the problems from the book until the student achieves mastery over the skill. An example of morph problem is shown in Figure 6. The Mastery Limit for the problems was set to three. Each problem used as Mastery Learning problem contained tutoring in the form of hints. Figure 5: An example of a problem from the book. An assignment in the Mastery Mode condition consisted of a mix of regular problems which would not cause the student to enter mastery mode upon an incorrect answer, and problems in which an incorrect answer would cause them to have to achieve mastery to progress. The regular problems gave tutoring in the form of hints. Once the students entered Mastery Mode, he got 50 chances to master the material before being presented the next problem in the homework. Figure 7 shows an example of hints in Mastery Mode. The posttest was three questions given as part of their normal homework. In fact these questions were simply questions that showed up on the students’ next homework assignment and came right from the text book, so it is not like we came up without posttest measures and then engineered the experiment to show this effect. Figure 6: A morph problem used in the Mastery Mode. Procedure The homework assignments were part of students’ regular schoolwork. The teachers taught the material in class and the homework was assigned to augment the schoolwork. Subsequently, the students did their homework using their home computers in the ASSISTments system. The next day, the teachers reviewed the material, progressed through the curriculum and assigned another set of homework problems. Students were encouraged by the teachers to spend up to 30 minutes doing the homework assignment. The students in the two conditions were assigned two different homework assignments. They were then tested on their proficiency for a certain skill by looking at three problems of the same skill in the following homework. The posttest was assigned 2 to 4 days after the pretest assignment was completed. The time interval allowed the posttest to be used as a delayed retention test. Figure 7: The morph problem in Figure 4 with hints expanded. Results Of the 128 students who participated in the study, 64 were assigned to the Mastery condition and the rest to the Non-Mastery condition. The perfect 50%-50% split in the participants was by chance as the computer randomly did the split. We only included the data of 105 students who completed the posttest in our final analysis. The rest did not complete either part or all of the posttest problems. This way we prevent a possible selection effect that might result if we selected only students that finished the experiment as the two groups of students probably would have differential rates of finishing in the two conditions. Of the 105 students, 55 were in NonMastery mode and the remaining 50 in Mastery Mode. The average post test score was 2.41 with standard deviation of 0.77. The average score of students in Mastery mode was 2.64 while the average score of students in Non-Mastery mode was 2.20. There was a significant effect for condition, t(97.2) = 3.095, p = .003, with students in Mastery Mode receiving higher scores than students in Non-Mastery Mode. The effect size was 0.52 and we can say with 95% confidence that the effect size lies within 0.20 to 0.98. This is a significant effect in the direction of Mastery mode which means the students learn more when their regular homework assignments are coupled with Mastery Learning. Figure 8 shows the results of the experiment. In order to analyze the immediate learning effect of Mastery Learning, we looked at the performance of students in a problem with similar skill in the same homework assignment. The first assignment consists of three problems that deal with writing exponential equations given growth factor and start value. We looked at two such problems and compared proportions of students who get the first one wrong but the second one wrong. The proportions of students who go the first problem wrong but got in second problem right in the treatment and control groups are shown in Figure 9.